In this project, we're going to work with data from the Survey of Consumer Finances (SCF). The SCF is a survey sponsored by the US Federal Reserve. It tracks financial, demographic, and opinion information about families in the United States. The survey is conducted every three years, and we'll work with an extract of the results from 2019.

1. Prepare Data¶

1.1. Import¶

First, we need to load the data, which is stored in a compressed CSV file: SCFP2019.csv.gz. In the last project, you learned how to decompress files using gzip and the command line. However, pandas read_csv function can work with compressed files directly.

Task 6.1.1: Read the file "data/SCFP2019.csv.gz" into the DataFrame df.

• Read a CSV file into a DataFrame using pandas.

One of the first things you might notice here is that this dataset is HUGE — over 20,000 rows and 351 columns! SO MUCH DATA!!! We won't have time to explore all of the features in this dataset, but you can look in the data dictionary for this project for details and links to the official Code Book. For now, let's just say that this dataset tracks all sorts of behaviors relating to the ways households earn, save, and spend money in the United States.

For this project, we're going to focus on households that have "been turned down for credit or feared being denied credit in the past 5 years." These households are identified in the "TURNFEAR" column.

Task 6.1.2: Use amask to subset create df to only households that have been turned down or feared being turned down for credit ("TURNFEAR" == 1). Assign this subset to the variable name df_fear.

• Subset a DataFrame with a mask using pandas.

1.2. Explore¶

1.2.1. Age¶

Now that we have our subset, let's explore the characteristics of this group. One of the features is age group ("AGECL").

Task 6.1.3: Create a list age_groups with the unique values in the "AGECL" column. Then review the entry for "AGECL" in the Code Book to determine what the values represent.

• Determine the unique values in a column using pandas.

Looking at the Code Book we can see that "AGECL" represents categorical data, even though the values in the column are numeric.

This simplifies data storage, but it's not very human-readable. So before we create a visualization, let's create a version of this column that uses the actual group names.

Task 6.1.4: Create a Series agecl that contains the observations from "AGECL" using the true group names.

• Create a Series in pandas.
• Replace values in a column using pandas.

Now that we have better labels, let's make a bar chart and see the age distribution of our group.

Task 6.1.5: Create a bar chart showing the value counts from age_cl. Be sure to label the x-axis "Age Group", the y-axis "Frequency (count)", and use the title "Credit Fearful: Age Groups".

• Create a bar chart using pandas.

You might have noticed that by creating their own age groups, the authors of the survey have basically made a histogram for us comprised of 6 bins. Our chart is telling us that many of the people who fear being denied credit are younger. But the first two age groups cover a wider range than the other four. So it might be useful to look inside those values to get a more granular understanding of the data.

To do that, we'll need to look at a different variable: "AGE". Whereas "AGECL" was a categorical variable, "AGE" is continuous, so we can use it to make a histogram of our own.

Task 6.1.6: Create a histogram of the "AGE" column with 10 bins. Be sure to label the x-axis "Age", the y-axis "Frequency (count)", and use the title "Credit Fearful: Age Distribution".

• Create a histogram using pandas.

It looks like younger people are still more concerned about being able to secure a loan than older people, but the people who are most concerned seem to be between 30 and 40.

1.2.2. Race¶

Now that we have an understanding of how age relates to our outcome of interest, let's try some other possibilities, starting with race. If we look at the Code Book for "RACE", we can see that there are 4 categories.

Note that there's no 4 category here. If a value for 4 did exist, it would be reasonable to assign it to "Asian American / Pacific Islander" — a group that doesn't seem to be represented in the dataset. This is a strange omission, but you'll often find that large public datasets have these sorts of issues. The important thing is to always read the data dictionary carefully. In this case, remember that this dataset doesn't provide a complete picture of race in America — something that you'd have to explain to anyone interested in your analysis.

Task 6.1.7: Create a horizontal bar chart showing the normalized value counts for "RACE". In your chart, you should replace the numerical values with the true group names. Be sure to label the x-axis "Frequency (%)", the y-axis "Race", and use the title "Credit Fearful: Racial Groups". Finally, set the xlim for this plot to (0,1).

• Create a bar chart using pandas.

This suggests that White/Non-Hispanic people worry more about being denied credit, but thinking critically about what we're seeing, that might be because there are more White/Non-Hispanic in the population of the United States than there are other racial groups, and the sample for this survey was specifically drawn to be representative of the population as a whole.

Task 6.1.8: Recreate the horizontal bar chart you just made, but this time use the entire dataset df instead of the subset df_fear. The title of this plot should be "SCF Respondents: Racial Groups"

• Create a bar chart using pandas.

How does this second bar chart change our perception of the first one? On the one hand, we can see that White Non-Hispanics account for around 70% of whole dataset, but only 54% of credit fearful respondents. On the other hand, Black and Hispanic respondents represent 23% of the whole dataset but 40% of credit fearful respondents. In other words, Black and Hispanic households are actually more likely to be in the credit fearful group.

1.2.3. Income¶

What about income level? Are people with lower incomes concerned about being denied credit, or is that something people with more money worry about? In order to answer that question, we'll need to again compare the entire dataset with our subgroup using the "INCCAT" feature, which captures income percentile groups. This time, though, we'll make a single, side-by-side bar chart.

Task 6.1.9: Create a DataFrame df_inccat that shows the normalized frequency for income categories for both the credit fearful and non-credit fearful households in the dataset. Your final DataFrame should look something like this:

    TURNFEAR   INCCAT  frequency
0          0   90-100   0.297296
1          0  60-79.9   0.174841
2          0  40-59.9   0.143146
3          0     0-20   0.140343
4          0  21-39.9   0.135933
5          0  80-89.9   0.108441
6          1     0-20   0.288125
7          1  21-39.9   0.256327
8          1  40-59.9   0.228856
9          1  60-79.9   0.132598
10         1   90-100   0.048886
11         1  80-89.9   0.045209

• Aggregate data in a Series using value_counts in pandas.
• Aggregate data using the groupby method in pandas.
• Create a Series in pandas.
• Rename a Series in pandas.
• Replace values in a column using pandas.
• Set and reset the index of a DataFrame in pandas.

Task 6.1.10: Using seaborn, create a side-by-side bar chart of df_inccat. Set hue to "TURNFEAR", and make sure that the income categories are in the correct order along the x-axis. Label to the x-axis "Income Category", the y-axis "Frequency (%)", and use the title "Income Distribution: Credit Fearful vs. Non-fearful".

• Create a bar chart using seaborn.

Comparing the income categories across the fearful and non-fearful groups, we can see that credit fearful households are much more common in the lower income categories. In other words, the credit fearful have lower incomes.

So, based on all this, what do we know? Among the people who responded that they were indeed worried about being approved for credit after having been denied in the past five years, a plurality of the young and low-income had the highest number of respondents. That makes sense, right? Young people tend to make less money and rely more heavily on credit to get their lives off the ground, so having been denied credit makes them more anxious about the future.

1.2.4. Assets¶

Not all the data is demographic, though. If you were working for a bank, you would probably care less about how old the people are, and more about their ability to carry more debt. If we were going to build a model for that, we'd want to establish some relationships among the variables, and making some correlation matrices is a good place to start.

First, let's zoom out a little bit. We've been looking at only the people who answered "yes" when the survey asked about "TURNFEAR", but what if we looked at everyone instead? To begin with, let's bring in a clear dataset and run a single correlation.

Task 6.1.11: Calculate the correlation coefficient for "ASSET" and "HOUSES" in the whole dataset df.

• Calculate the correlation coefficient for two Series using pandas.

That's a moderate positive correlation, which we would probably expect, right? For many Americans, the value of their primary residence makes up most of the value of their total assets. What about the people in our TURNFEAR subset, though? Let's run that correlation to see if there's a difference.

Task 6.1.12: Calculate the correlation coefficient for "ASSET" and "HOUSES" in the whole credit-fearful subset df_fear.

• Calculate the correlation coefficient for two Series using pandas.

Aha! They're different! It's still only a moderate positive correlation, but the relationship between the total value of assets and the value of the primary residence is stronger for our TURNFEAR group than it is for the population as a whole.

Let's make correlation matrices using the rest of the data for both df and df_fear and see if the differences persist. Here, we'll look at only 5 features: "ASSET", "HOUSES", "INCOME", "DEBT", and "EDUC".

Task 6.1.13: Make a correlation matrix using df, considering only the columns "ASSET", "HOUSES", "INCOME", "DEBT", and "EDUC".

• Create a correlation matrix in pandas.

Task 6.1.14: Make a correlation matrix using df_fear.

• Create a correlation matrix in pandas.

Whoa! There are some pretty important differences here! The relationship between "DEBT" and "HOUSES" is positive for both datasets, but while the coefficient for df is fairly weak at 0.26, the same number for df_fear is 0.96.

Remember, the closer a correlation coefficient is to 1.0, the more exactly they correspond. In this case, that means the value of the primary residence and the total debt held by the household is getting pretty close to being the same. This suggests that the main source of debt being carried by our "TURNFEAR" folks is their primary residence, which, again, is an intuitive finding.

"DEBT" and "ASSET" share a similarly striking difference, as do "EDUC" and "DEBT" which, while not as extreme a contrast as the other, is still big enough to catch the interest of our hypothetical banker.

Let's make some visualizations to show these relationships graphically.

1.2.5. Education¶

First, let's start with education levels "EDUC", comparing credit fearful and non-credit fearful groups.

Task 6.1.15: Create a DataFrame df_educ that shows the normalized frequency for education categories for both the credit fearful and non-credit fearful households in the dataset. This will be similar in format to df_inccat, but focus on education. Note that you don't need to replace the numerical values in "EDUC" with the true labels.

    TURNFEAR  EDUC  frequency
0          0    12   0.257481
1          0     8   0.192029
2          0    13   0.149823
3          0     9   0.129833
4          0    14   0.096117
5          0    10   0.051150
...
25         1     5   0.015358
26         1     2   0.012979
27         1     3   0.011897
28         1     1   0.005408
29         1    -1   0.003245

• Aggregate data in a Series using value_counts in pandas.
• Aggregate data using the groupby method in pandas.
• Create a Series in pandas.
• Rename a Series in pandas.
• Replace values in a column using pandas.
• Set and reset the index of a DataFrame in pandas.

Task 6.1.16: Using seaborn, create a side-by-side bar chart of df_educ. Set hue to "TURNFEAR", and make sure that the education categories are in the correct order along the x-axis. Label to the x-axis "Education Level", the y-axis "Frequency (%)", and use the title "Educational Attainment: Credit Fearful vs. Non-fearful".

• Create a bar chart using seaborn.

In this plot, we can see that a much higher proportion of credit-fearful respondents have only a high school diploma, while university degrees are more common among the non-credit fearful.

1.2.6. Debt¶

Let's keep going with some scatter plots that look at debt.

Task 6.1.17: Use df to make a scatter plot showing the relationship between DEBT and ASSET.

• Create a scatter plot with pandas.

Task 6.1.18: Use df_fear to make a scatter plot showing the relationship between DEBT and ASSET.

• Create a scatter plot with pandas.

You can see relationship in our df_fear graph is flatter than the one in our df graph, but they clearly are different.

Let's end with the most striking difference from our matrices, and make some scatter plots showing the difference between HOUSES and DEBT.

Task 6.1.19: Use df to make a scatter plot showing the relationship between HOUSES and DEBT.

• Create a scatter plot with pandas.

And make the same scatter plot using df_fear.

Task 6.1.20: Use df_fear to make a scatter plot showing the relationship between HOUSES and DEBT.

• Create a scatter plot with pandas.

The outliers make it a little difficult to see the difference between these two plots, but the relationship is clear enough: our df_fear graph shows an almost perfect linear relationship, while our df graph shows something a little more muddled. You might also notice that the datapoints on the df_fear graph form several little groups. Those are called "clusters," and we'll be talking more about how to analyze clustered data in the next lesson.

Copyright © 2022 WorldQuant University. This content is licensed solely for personal use. Redistribution or publication of this material is strictly prohibited.

6.2. Clustering with Two Features

In the previous lesson, you explored data from the Survey of Consumer Finances (SCF), paying special attention to households that have been turned down for credit or feared being denied credit. In this lesson, we'll build a model to segment those households into distinct clusters, and examine the differences between those clusters.

1. Prepare Data¶

1.1. Import¶

Just like always, we need to begin by bringing our data into the project. We spent some time in the previous lesson working with a subset of the larger SCF dataset called "TURNFEAR". Let's start with that.

Task 6.2.1: Create a wrangle function that takes a path of a CSV file as input, reads the file into a DataFrame, subsets the data to households that have been turned down for credit or feared being denied credit in the past 5 years (see "TURNFEAR"), and returns the subset DataFrame.

• Write a function in Python.
• Subset a DataFrame by selecting one or more columns in pandas.

And now that we've got that taken care of, we'll import the data and see what we've got.

Task 6.2.2: Use your wrangle function to read the file SCFP2019.csv.gz into a DataFrame named df.

• Read a CSV file into a DataFrame using pandas.

1.2. Explore¶

We looked at a lot of different features of the "TURNFEAR" subset in the last lesson, and the last thing we looked at was the relationship between real estate and debt. To refresh our memory on what that relationship looked like, let's make that graph again.

Task 6.2.3: Create a scatter plot of that shows the total value of primary residence of a household ("HOUSES") as a function of the total value of household debt ("DEBT"). Be sure to label your x-axis as "Household Debt", your y-axis as "Home Value", and use the title "Credit Fearful: Home Value vs. Household Debt".

• What's a scatter plot?
• Create a scatter plot using seaborn.

Remember that graph and its clusters? Let's get a little deeper into it.

1.3. Split¶

We need to split our data, but we're not going to need target vector or a test set this time around. That's because the model we'll be building involves unsupervised learning. It's called unsupervised because the model doesn't try to map input to a st of labels or targets that already exist. It's kind of like how humans learn new skills, in that we don't always have models to copy. Sometimes, we just try out something and see what happens. Keep in mind that this doesn't make these models any less useful, it just makes them different.

So, keeping that in mind, let's do the split.

Task 6.2.4: Create the feature matrix X. It should contain two features only: "DEBT" and "HOUSES".

• What's a feature matrix?
• Subset a DataFrame by selecting one or more columns in pandas.

2. Build Model¶

Before we start building the model, let's take a second to talk about something called KMeans.

Take another look at the scatter plot we made at the beginning of this lesson. Remember how the datapoints form little clusters? It turns out we can use an algorithm that partitions the dataset into smaller groups.

Let's take a look at how those things work together.

Task 6.2.5: Run the cell below to display the ClusterWidget.

• What's a centroid?
• What's a cluster?

Take a second and run slowly through all the positions on the slider. At the first position, there's whole bunch of gray datapoints, and if you look carefully, you'll see there are also three stars. Those stars are the centroids. At first, their position is set randomly. If you move the slider one more position to the right, you'll see all the gray points change colors that correspond to three clusters.

Since a centroid represents the mean value of all the data in the cluster, we would expect it to fall in the center of whatever cluster it's in. That's what will happen if you move the slider one more position to the right. See how the centroids moved?

Aha! But since they moved, the datapoints might not be in the right clusters anymore. Move the slider again, and you'll see the data points redistribute themselves to better reflect the new position of the centroids. The new clusters mean that the centroids also need to move, which will lead to the clusters changing again, and so on, until all the datapoints end up in the right cluster with a centroid that reflects the mean value of all those points.

Let's see what happens when we try the same with our "DEBT" and "HOUSES" data.

Task 6.2.6: Run the cell below to display the SCFClusterWidget.

2.1. Iterate¶

Now that you've had a chance to play around with the process a little bit, let's get into how to build a model that does the same thing.

Task 6.2.7: Build a KMeans model, assign it to the variable name model, and fit it to the training data X.

• What's k-means clustering?
• Fit a model to training data in scikit-learn.

And there it is. 42 datapoints spread across three clusters. Let's grab the labels that the model has assigned to the data points so we can start making a new visualization.

Task 6.2.8: Extract the labels that your model created during training and assign them to the variable labels.

• Access an object in a pipeline in scikit-learn.

Using the labels we just extracted, let's recreate the scatter plot from before, this time we'll color each point according to the cluster to which the model assigned it.

Task 6.2.9: Recreate the "Home Value vs. Household Debt" scatter plot you made above, but with two changes. First, use seaborn to create the plot. Second, pass your labels to the hue argument, and set the palette argument to "deep".

• What's a scatter plot?
• Create a scatter plot using seaborn.

Nice! Each cluster has its own color. The centroids are still missing, so let's pull those out.

Task 6.2.10: Extract the centroids that your model created during training, and assign them to the variable centroids.

• What's a centroid?

Let's add the centroids to the graph.

Task 6.2.11: Recreate the seaborn "Home Value vs. Household Debt" scatter plot you just made, but with one difference: Add the centroids to the plot. Be sure to set the centroids color to "gray".

• What's a scatter plot?
• Create a scatter plot using seaborn.

That looks great, but let's not pat ourselves on the back just yet. Even though our graph makes it look like the clusters are correctly assigned but, as data scientists, we need a numerical evaluation. The data we're using is pretty clear-cut, but if things were a little more muddled, we'd want to run some calculations to make sure we got everything right.

There are two metrics that we'll use to evaluate our clusters. We'll start with inertia, which measure the distance between the points within the same cluster.

Task 6.2.12: Extract the inertia for your model and assign it to the variable inertia.

• What's inertia?
• Access an object in a pipeline in scikit-learn.
• Calculate the inertia for a model in scikit-learn.

The "best" inertia is 0, and our score is pretty far from that. Does that mean our model is "bad?" Not necessarily. Inertia is a measurement of distance (like mean absolute error from Project 2). This means that the unit of measurement for inertia depends on the unit of measurement of our x- and y-axes. And since "DEBT" and "HOUSES" are measured in tens of millions of dollars, it's not surprising that inertia is so large.

However, it would be helpful to have metric that was easier to interpret, and that's where silhouette score comes in. Silhouette score measures the distance between different clusters. It ranges from -1 (the worst) to 1 (the best), so it's easier to interpret than inertia. WQU WorldQuant University Applied Data Science Lab QQQQ

Task 6.2.13: Calculate the silhouette score for your model and assign it to the variable ss.

• What's silhouette score?
• Calculate the silhouette score for a model in scikit-learn.

Outstanding! 0.976 is pretty close to 1, so our model has done a good job at identifying 3 clusters that are far away from each other.

It's important to remember that these performance metrics are the result of the number of clusters we told our model to create. In unsupervised learning, the number of clusters is hyperparameter that you set before training your model. So what would happen if we change the number of clusters? Will it lead to better performance? Let's try!